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PROVISIONAL APPLICATION 

Inventors: John J. Keating III, Rainer Martini, 

Both residing at 613 Hudson Street, Hoboken NJ 07030 
Title: Image Scanner Realtime Calibration of Stereographic Images 

1 . Background of the Invention: In developing an invention disclosure for a 3D image 

scanner using ultra short pulses, it was realized that the calibration of the optical equipment, 
especially those configurations using multiple cameras is a very time consuming process and 
that the cost of three dimensional (3D) imaging equipment could be reduced if a more 
efficient calibration equipment were available. This invention employs a holographic 
projection to which multiple image recorders are aligned by physical alignment or by software 
post processing of the images produced by stereoscopic imaging capturing devices. 

2. Purposes and advantages of invention: This invention provides apparatus and methods to 
calibrate stereoscopic optical recording equipment, in realtime, for both optical and electronic 
based optical recorders. Stereoscopic Equipment includes two or more optical recorders or a 
single optical recorder sequentially moved to different positions in space to acquire images of 
one or more desired objects. Calibration Equipment improves the quality of the stereoscopic 
images, for example, by more accurately combining information from multiple Optical 
Recording Devices. The herein described Calibration Equipment is both an apparatus to use 
with image capture equipment such as cameras and a method to compensate for the 
absolute and/or differential distortion within and across stereoscopic imaging equipment used 
together as a system. 

One potential use of the higher quality images obtained from calibration/compensation is 
to more accurately determine the distance to points on the stereoscopically recorded or 
viewed object for uses which include the construction of three dimensional images of desired 
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object. As this equipment is capable of providing calibration during normal stereoptic 
operation, without affecting the desired image, it is usable for continuous calibration, for 
example, when the stereoptic imaging devices are equipped with variable zoom lenses that 
exhibit differing optical properties as the zoom setting is changed or continuous calibration 
during normal operation when mechanically misalignment of equipment occurs in one or 
more of the optical paths to the Optical Recording Devices. 

3. Description: The calibration of the three dimensional stereoscopic imaging equipment 
normally requires the calibration for all points in the target space occupied by the Desired 
Object to be captured as a three dimensional image, a time consuming process that is 
repeated for each change of the equipment in the optical path between the recorded objects 
and the image recording devices. To overcome this problem this disclosure describes an 
apparatus employing a holographic projection technique that creates a virtual object in the 
target space and therefore automates the calibration process. As this virtual image is viewed 
by the same lens system used for the stereoptic imaging systems, all static distortion will be 
taken into account. Changing the Holographic Calibration Pattern (image) is accomplished 
by recording multiple, superimposed holograms at different wavelength(s) or different 
positions on the Holographic Calibration Plate and then choosing a specific calibration pattern 
by illuminating the Holographic Calibration Plate by the specific wavelength(s) corresponding 
to the specific calibration pattern or viewing from a specific position. This apparatus and 
method allows for calibration at selectable and assignable positions in the Desired Object 
three dimensional target space. The hologram allows explicit calibration in the target space 
without the limitations of using real calibration objects that are moved through the image 
space as a calibration method. This holographic calibration system envisions the use of fixed 
holographic images or variable plate (e.g. using external inputs to phase modulate the 
holographic plate) holograms. 

4. Physical Arrangement: 
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In Figure 1, starting with the three dimensional physical object with normal background 
illumination, such as the three dimension pyramid shown in Figure 1. reflected light from the 
object travels toward the stereoscopic system, first passing through a Holographic Calibration 
Plate that is transparent to light in the desired range (e.g., normal visible light reflected from 
the Desired Object, which in Figure 1 is a pyramid, so as not to effect the recording of light of 
the Desired Wavelengths by the Optical Recorder(s). Let us refer to the light from the 
Desired Object whose image is desired( e.g. the pyramid in Figure 1) as the Desired 
Wavelengths. Let us refer the wavelengths produced the Holographic Calibration Plate, the 
Virtual Calibration Object as the Calibration Wavelengths. The Desired Wavelengths passing 
through the Holographic Calibration Plate [abed] which will contain holographic encoded 
calibration information. This calibration information is in the Desired Wavelength region for 
applications in which the calibration information overlays the Desired Wavelength information 
so that both the Desired Object and the hologram be simultaneously observable without 
additional post processing. For applications in which the Desired Wavelengths alone are 
desired, the Calibration Wavelengths are stored or processed separately from the Desired 
Wavelength information. Calibration information and Desired information at overlapping 
wavelengths can also be temporally separated by pulsing the Calibration information for short 
times and synchronizing observation of the Calibration information. 

The calibration patterns are encoded into the calibration hologram and made recordable and 
storable by the optical sensors by illuminating the hologram [abed] in Figure 1 with an 
independent light source. S1 in Figure 1. which may be on either side of the hologram, thus 
ensuring the incorporation of the Holographic Calibration image into the camera field of view. 
Through movement of holographic plate or by changing of the illumination parameters (e.g. 
frequency of S1), alternative calibration information is generated for processing by the Optical 
Recorder. 

Figure 1a is a variation of Figure 1 that uses a Mirror M1 to reflect the holographic image 
from Hologram M1, e.g. correction image, to the one or more Optical Recorders R1, R2, and 
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R3. The mirror M1 is reflective at the calibration wavelengths and transparent at the desired 
wavelengths, thus images from the desired object, e.g. the pyramid in Figure 1a, pass directly 
to the Optical Recorders. This variation enables the use of wavelengths in light source S1 
that included the desired wavelengths, as the desired wavelengths from S1 will pass directly 
through the Mirror M1 and are not reflected to the Optical Recorders. 

In the case where the Holographic Calibration Wavelengths are different from the Desired 
Wavelengths (the normal case for this apparatus, although some applications may choose a 
Calibration Wavelength within the range of the Desired Wavelengths), the images reaching 
the Optical Recorders Device shown in Figure 1 contains both the Desired Object 
Wavelengths and Calibration Wavelengths. 

The Optical Recorder equipment is this disclosure is of two types: 

1. For Optical Recorder Device (apparatus), shown in Figure 2, the composite image of 
Desired and Calibration Wavelengths enters each Optical Recorder Device and is 
separated inside the Optical Recorder using additional wavelength selective filters 
and the Desired Wavelengths and the Calibration Wavelengths are processed 
separately within the Optical Recording Device. Selective separation of the optical 
wavelengths is of multiple subtypes. Type 1a uses optical filters to selectively 
separate the wavelengths. Type lb uses a special optical to electron converts such 
as CCD arrays to separate the wavelengths, that is, the pixel array that provides 
optical to electronic conversion uses a planar arrangement of light sensitive devices 
whose arrangement in two dimensions alternates devices whose sensitivity peaks for 
Desired and Calibration Wavelengths. Type 1c splits the optical signal and optical 
band pass and Bandstop filters select the Desired Wavelengths and the Calibration 
Wavelengths. 
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2. In Type 2 Optical Recorder equipment, shown in Figure 3. the calibration image 
wavelengths are generated only for special frames. Frames imply that the recording 
of the desired and calibration optical images is broken into discrete units of time 
during which subsequent samples of the Desired Wavelengths and Calibration 
Wavelengths are captured. Special frames store the Calibration Wavelengths. The 
special frames are of Type A which are stored or processed separately from the 
desired frames or Type B which are interspersed, at a periodic or non periodic rate, 
between the frames that contain the Desired Wavelengths. Figure 3 shows the 
equipment that utilizes Type 2 Optical Recorders. In Figure 3, Hologram Source S2 is 
pulsed on during the special frames and the optical or mechanical shutter 0S1 is 
closed during special frames. This arrangement enables the recording of desired 
and calibration information without the wavelength separation described in Figure 1 . 
The apparatus in Figure 3 more easily enables multiple calibration patterns based on 
Calibration Wavelengths above and below the Desired Wavelengths than the 
apparatus described above in Figure 1 in which additional wavelength selective filters 
are required. 

The Type 2 apparatus permits a pulsed synchronization source in the field of 
view to provide pulsed time code information for synchronizing multiple cameras 
and/or to convey synchronized instructions to the multiple stereoptic cameras 
simultaneously, e.g. for special effects and other camera related functions. The 
combination of synchronization and holographic calibration/alignment across multiple 
cameras permits a more cost effective implementation of panoramic cameras such 
as those that are now achieved mechanically in CineMax systems and the simplified 
construction of panoramic three dimensional imaging devices. 



The apparatus described above in Figures 1, 1a, 2 and 3 generates, in each Optical 
Recorder in the system, an image of the Desired Object and an image of the Virtual 
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Calibration Object. For each mechanical configuration of the Optical Recorders, e.g. 
physical position or optical magnification setting, the calibration method is a follows: 

1 . Choose one Optical Recorder as the reference and assign a coordinate system. For 
an electronic image detector within the Optical Recorder assign the coordinates in 
parallel or normal to the pixel array in the camera or in alignment with the 
Holographic Calibration Plate pattern. For a non-electronic system, assign the 
coordinate system arbitrarily or in alignment with the holographic calibration pattern. 

2. Assign the same coordinate system to all other Optical Recorders and measure the 
difference in the hologram calibration pattern in this coordinate system. 

3. Utilize the differences measured on this coordinate system to calculate the calibration 
corrections for each Optical Recorder relative to the reference Optical Recorder. 

4. Use the calibration corrections to compensate the desired images either 
mechanically or electronically. 

When a single Optical Recorder is used sequentially in different spatial positions, the 
above procedure changes as follows: The first position becomes the reference and each 
subsequent position is treated in the calibration method as an additional Optical 
Recorder. 

The Optical Recorder Device, shown in Figure 2, is capable of processing the image of a 
calibrated desired object in real time by translating, rotating and scaling the Desired 
Object Image based on the Calibration Image or storing the Calibration Images and 
Desired Object Images independently for delayed processing. 

The Holographic Calibration Plate [abed] in the Figures 1 and 3 need not be flat or 
continuously connected. A single Optical Recorder or multiple Optical Recorders may 
use a spherical, cylindrical or arbitrarily shaped hologram, illuminated by Source S1 from 
either the inside or outside of the plate that surrounds or partially surrounds the Desired 
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Object As an example. Figure 4 shows a single Optical Recorder that moves around the 
outside of the cylindrical holographic calibration plate and generates successive images 
of the object that are post processed into a stereoptic image or into a three dimensional 
model of the object using algorithms for color and or edge matching. These stereoptic 
images of the object can be used to capture the 3 dimensional shape of the object from 
calculation using as input successive images of the Desired Object, and the spatial 
position of uniquely determined points on the object, e.g. points for a wire frame model, is 
calculated by triangulation. Color, texture and shading are then applied to the wire frame 
model from the captured images of the object. 

Specific points on the objects are uniquely determined by highlight the desired points with 
points painted, points where an application pattern is projected on to the object using a 
calibration wavelength or with calibration points attached to the object which are painted 
with the calibration wavelengths. For example, to use this apparatus to measure a 
person for shoes or clothing, the person puts on a pair or socks or a tights that are 
painted with an application pattern, e.g. grid, in one or more calibration wavelengths. The 
image matching, for one or more cameras, is precise at the calibration grid points. 
Alternatively, a cosmetic surgeon may paint a selected pattern on the patient using paint 
in a calibration wavelength that is not visible to the human eye, or project a pattern on the 
patient using calibration wavelengths. This allows the surgeon to view normal visible 
images of the patient that have the selected pattern visible in post processing for either 
still or full motion images. Surgeons may also use these techniques in conjunction with 
thermal images when the desired wavelengths are properly selected, for radiological 
treatment application where heat is generated by radioactive treatment materials. 
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Optical Recorders which separate the desired and Calibration Wavelengths enable 
efficient extraction of the objects color and texture in addition to its three dimensional 
shape. Figure 5 shows the use of three Optical Recorders which can capture and 
triangulate the position in three dimensional space of a point P which is illuminated by a 
laser using a Calibration Wavelength, and thus invisible at Desired Wavelenths. This is 
an improvement over stereoptic color matching or edge matching, as the point P is 
precise. Furthermore', when combined with holographic calibration techniques, this 
apparatus improves the ease and accuracy of the three dimensional shape resolutions 
that is the position of point P can be inferred from its position relative to the Virtual 
Holographic Pattern. As the processing of the object color and texture information is 
independent of the spatial information this technique this apparatus captures both the 
objects shape and texture. Combining a holographic measurement grid with the point in 
illuminated at a Calibration Wavelength provides an apparatus that can measure an 
objects shape with a single Optical Recorder and provide increased speed and accuracy 
with multiple Optical Recorders. 

The apparatus uses the Virtual Calibration Pattern to position the Laser Pointer, L1 at 
desired points on the Virtual Calibration Pattern. For example, if the Virtual Calibration 
Pattern is a grid on a plane that intersects the Desired Object and the laser generated 
point P is to be positioned on the surface of the object at all points at the grid intersection 
points nearest to the spatial positions where the virtual grid intersects the Desired Object, 
the following procedure is followed; 

1 . Choose a Virtual Calibration Pattern grid on a plane that is tangent to the nearest 
point of the Desired Object as measured from the Optical Recorder 

2. Label the subset of virtual grid intersection points defined by those closest to where 
the virtual grid intersects the desired object in some numerical order. 
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3. Starting with the first numbered point and continuing to the last numbered point, 
position the laser point P at each numbered point successively and collect the 
position data using measurements to the calibration pattern. The apparatus 
simplifies the positioning process since the laser pointer wavelength is a Calibration 
Pattern Wavelength and the Pointer position P and the Virtual Calibration Pattern are 
simultaneously observable by the control system for the laser pointer and thus 
relative positioning corrections rather than absolute position Laser Pointing system 
are required. 

4. Generate successive Virtual Calibration Pattern grids that intersect the Desired 
Object at greater distances from the Optical Recorder and repeat steps 2 and 3. 

5. Continue until the Virtual Calibrating Pattern girds no longer intersect the Virtual 
Object 

Figure 6 shows the combination of the holographic calibration with a Laser Ranging 
Measurement Device LRMD1. This apparatus combines a Laser Ranging Measurement 
Device with holographic three dimensional calibration using one or more Optical 
Recorders. The Laser Ranging Measurement Device LRMD1 provides accurate ranges 
to any point P on the surface of the Desired Object. The holographic calibration grid is 
used to provide accurate measurements of point P relative to optical images gathered 
from the Desired Object. By choosing a wavelength for the Laser Ranging Measurement 
Device LRMD1 that is a Calibration Wavelength, the point P illuminated by the LRMD1 is 
observable at the same time as the calibration pattern. The Laser Ranging Measurement 
Device is positioned using the procedure described with reference to Figure 5. 

Figure 7 shows the use of a spherical calibration hologram that generates a calibration 
grid, e.g. a spherical coordinate grid, around one or more Optical Recording Devices. 
Apparatus in this configuration are well suited to applications requiring a complete field of 
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view. Multiple Optical Recorders can be placed within the calibration hologram to 
simultaneously cover all directions although this requires appropriate handling of 
overlapping fields of vision between adjacent Optical Recording Devices. The Calibration 
Hologram provides the mechanism to overcome the problems of adjacent Optical 
Recording Devices, including overlapping fields and conformal mapping from the planar 
segment of the optical recording device to a spherical frame of reference, and also 
provides a holographic pattern to provide the basis for conformal mapping of the Optical 
Recorder Outputs. The holographic calibration pattern provides boundaries between 
adjacent Optical Recording Devices and provides a uniform coordinate system across all 
devices that enables simplified calibration and alignment of conformal mapping across 
Optical Recording Devices, especially when the alignment between such devices are 
varying due to shock, vibration or acceleration. The Spherical Calibration Hologram, 
when combined with the laser ranging of Figure 6 provides accurate ranging to a targeted 
object. 

This spherical holographic calibration configuration provides an apparatus to generate an 
input for a panoramic 360 degree view of remotely piloted vehicles subject to shock or 
acceleration. The apparatus enables realtime compensation for the two major problems 
of providing a panoramic 360 degree view for remotely piloted vehicles, that is, 
continuous alignment of multiple "fisheye" cameras as such cameras and related optics 
are subject to misalignment by shock or vibration, and the conformal mapping of multiple 
cameras into a 380 degree panoramic view. Furthermore the Calibration Wavelengths 
can be chosen to include wavelength used in collision avoidance systems and thus 
process such information jointly with the Optical Recorder calibration. 

Figure 8 shows the use of non continuous, identical holographic calibration plates. If these 
plates were held mechanically parallel, then this configuration would effectively be the 
configuration of Figure 1. Misalignment of the calibration plates shifts the calibration pattern 
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up or down, left or right and/or forward or back. Thus the misalignment may be calculated 
from a set of reference points in the field of view. These points my be known calibration 
points, such as fixed points in the field of view, points projected by a laser whose wavelength 
is a Calibration Wavelength or some distinctive feature of the Desired Object such as an 
edge or point of distinct color point. The calibration algorithm is as follows: 

1 . Fix the separate holographic correction plates rigidly as closely as possible to the 
configuration of single fixed plate For flat calibration plates this implies initially fixing the 
non-contiguous plates as parallel to each other as possible. For cylindrical/spherical 
segmented non-contiguous plates this implies initially fixing the plates rigidly as close the 
location which a contiguous cylindrical/spherical plate would occupy, and so forth for 
other shapes of holographic plates such as elliptical. 

2. Determine a number of reference points in the vicinity of the Desired Object, if not on the 
Desired Object itself. Illuminated points on the Desired Object are a subset of these 
conditions. Additional fixed calibration points are generated by affixing reflecting and/or 
absorbing colors to the Desired Object with wavelengths in the calibration range, thus 
predetermining a fixed set of points for calibration. 

3. For each reference point, determine the corresponding position on the Holographic 
Compensation Pattern and determine the misalignment of the hologram. 

4. Determine the correction factors, for example, shift, rotation and scaling in an orthogonal 
coordinate system, as a function of position in the Desired Object in three Dimensional 
space, for each Optical Recording Device. 

5. Applied the corrections for each Optical Recording Device to both the Holographic 
Calibration Patterns and the Desired Objects. 



An alternative configuration of the apparatus is realizable, for applications that do not utilize 
the patterns of the Holographic Correction Plate. Specific reference points in the field of view 
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of the Desired Object are illuminated by a remote source or self illuminated with Calibration 
Wavelengths. These points are separated into Calibration Wavelengths by the apparatus 
and are used to provide calibration across the Optical Recorders and provide the calibration 
for compensation of the Desired Object. That is, in some circumstances, fixed calibration 
points near or on the desired object may replace the Holographic Calibration Plate. 

For applications that do not utilize the patterns of the Holographic Calibration Plate, and 
where specific reference points can be illuminated by a remote source or self illuminated with 
Calibration Wavelengths the holographic plate is not a required part of the apparatus. Only 
the parts of the apparatus that separates the Desired Object wavelengths from Calibration 
Wavelengths is required. The calibration corrections for each Optical Recorder are 
calculated from the known points illuminated by the Calibration Wavelengths for each of the 
stereoptic Optical Recorders and calibration corrections are applied to each of the Desired 
Object in each Optical Recorder. This simplified calibration and compensation method is 
more likely to be employed only in wide angle images that uniformly capture many fixed 
calibration sources and where the fixed points are affixed to Desired Objects or placed near 
to stationary Desired Objects. 

Figure 9 shows the addition of a Bandstop Filter to the apparatus of Figure 1, which prevents 
the illumination wavelength(s) of Hologram Source(s) from traveling from the apparatus to the 
region in the vicinity of the Desired Object. Such emanations from the apparatus are an 
undesirable circumstance when the apparatus is used in a security or a surveillance 
application. 

Figure 10 show an apparatus which generates virtual holograms with stereoptic microscopes. 
While the calibration of the two optical paths in a microscope is not important due to the 
excellent mechanical construction of microscopes, the calibration is more useful when 
multiple optical paths are employed and in addition, the ability to project a hologram is both 
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novel and useful. Potential uses include projecting grids to assist in counting specimens such 
as while blood cells on microscope slides or projecting three dimension grids to ascertain the 
locations of imperfections in diamonds as a means of grading and identification in the case of 
theft. The use of three dimensional holograms permits improved analysis as stereoptic 
microscope images are scaled, especially in the depth dimension. 

When multiple holographic patterns are recorded on the Holographic Calibration Plate, these 
patters are selectively accessed as the wavelength of the illumination of the Holographic 
Calibration Plate is varied or by movement of the plate. One problem with a grid is that 
unless the grid intersections are somehow labeled, errors in identifying a particular 
intersection are possible. This apparatus uses one Calibration Wavelength to record the grid 
and another Calibration Wavelength to symbolically identify the grid intersection, e.g. a bar 
code or alphanumeric sequence, and additional wavelengths with finer grid structures for 
more accurate determination of three dimensional spatial positioning. Combining the 
Holographic Calibration Plate with a symbolic identification such a bar code decreases the 
time for real time recognition of grid intersections by commercial software. 



5. Rannifications: This invention provides continuous (realtime) calibration of stereo optic 
information from still or moving objects. It improves the quality of the stereo optic images and 
permits the construction of more accurate three dimensional models from such images. 

By changing the calibration patterns, for example by varying the hologram frequency when 
alternative holograms are recorded at different frequencies, several ranges and positions of 

* 

the image field of view can be continuously recalibrated to compensate for variations in 
image system distortions (e.g. as the degree of telephoto magnification is changed). 
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Compared to standard calibration systems this technique offers many possibilities. In a standard 
calibration procedure a real object has to be brought in all possible positions to obtain a full 
calibration - which is typically very time consuming and some positions are not accessible at all - 
and therefore cannot be calibrated. As consequence the calibration is time consuming. Any 
changes to the recording apparatus after the calibration will void this calibration and reduce the 
accuracy of the 3D information. With this new apparatus, the Virtual Holographic Calibration 
Pattern can be produced in any place In the field of view of the optical recording devices and the 
calibration is done quickly. 

Compared with other calibration systems, the desired information and the calibration 
information are recorded through the same lens of each image recorder as the desired image 
is captured at the same time as the calibration image, and this enables real time calibration 
as post processing. 

The Calibration Equipment and methodology continuously operate without interfering with the 
Optical Recording Devices. 
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Type 2 Optical Rttcordsr Davico 
Figure 3 
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Sterographic 3 Dimension Extraction With Las r Ranging 
Figure 6 
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Non-contiguous hotograpKlo calituration plates 



Figure 8 
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